When Large Language Models (LLMs) are compressed using techniques such as quantization, the predominant way to demonstrate the validity of such techniques is by measuring the model's accuracy on various benchmarks.If the accuracies of the baseline model and the compressed model are close, it is assumed that there was negligible degradation in quality.However, even when the accuracy of baseline and compressed model are similar, we observe the phenomenon of flips, wherein answers change from correct to incorrect and vice versa in proportion.We conduct a detailed study of metrics across multiple compression techniques, models and datasets, demonstrating that the behavior of compressed models as visible to end-users is often significantly different from the baseline model, even when accuracy is similar.We further evaluate compressed models qualitatively and quantitatively using MT-Bench and show that compressed models are significantly worse than baseline models in this free-form generative task.Thus, we argue that compression techniques should also be evaluated using distance metrics.We propose two such metrics, KL-Divergence and flips, and show that they are well correlated.

Pre-trained language models (PLMs) have seen tremendous success in text classification (TC) problems in the context of Natural Language Processing (NLP). In many real-world text classification tasks, the class definitions being learned do not remain constant but rather change with time - this is known as concept shift. Most techniques for handling concept shift rely on retraining the old classifiers with the newly labelled data. However, given the amount of training data required to fine-tune large DL models for the new concepts, the associated labelling costs can be prohibitively expensive and time consuming. In this work, we propose a reformulation, converting vanilla classification into an entailment-style problem that requires significantly less data to re-train the text classifier to adapt to new concepts. We demonstrate the effectiveness of our proposed method on both real world & synthetic datasets achieving absolute F1 gains upto 7% and 40% respectively in few-shot settings. Further, upon deployment, our solution also helped save 75% of labeling costs overall.

You're wearing a mask, as encouraged by the Centers for Disease Control and Prevention and possibly by your local government during this coronavirus outbreak, and you want to use FaceID to open your iPhone or Android phone. Yes, says a Chinese researcher from Tencent's Xuanwu lab. But according to Apple, this is frowned upon big time and could affect the integrity of your phone. There are several hacks available online, and most will send the user to reregister for FaceID on the iPhone by covering the left and right side of their face separately. After the researcher from Tencent originally demonstrated and popularized this setup, many others copied and tweaked with videos on YouTube.

Every coin has two sides. We've been talking for so long about all the coming benefits of wireless smart homes, connected homes and connected offices. And it's true, there are many incredible advancements in this space. However, there is an important warning I want to discuss. Everyone must understand other side of the coin which is seldom discussed, loss of privacy.

Every time there is a big wireless, telecom or technology trade show, the big question I am always asked by the media as a telecom and wireless analyst, is simple. What was the key message or take away from the show? Last week, at the world's largest wireless trade show, Mobile World Congress 2018 in Barcelona, Spain, the answer was clear. First it is about 5G, with plenty of AI and IoT mixed in. Yes, our world is rapidly changing. So, what will 5G, AI and IoT do for us?

The virtual Google helper, titled Google Assistant, debuted alongside Allo--the company's messaging app. The app for Android and iOS not only allowed users to chat with friends, but also with a digital assistant that would fetch information in its own chat window, or even while you were in a chat with someone else. Instead of requiring users to speak out loud, the Google Assistant allows users to type out questions and responses. The above gif, hailing from Android Police, speculates that the updated version of Android will more closely integrate the recently revealed assistant. Google Assistant throughout Android would not only allow Google users to say "Ok, Google" to summon their assistant, but type to it too.